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Abstract. We present several formulas for the traces of elements in complex 
hyperbolic triangle groups generated by complex reflections. 

The space of such groups of fixed signature is of real dimension one. We 
parameterise this space by a real invariant a of triangles in the complex hy- 
perbolic plane. The main result of the paper is a formula, which expresses the 
trace of an element of the group as a Laurent polynomial in e'" with coeffi- 
cients independent of a and computable using a certain combinatorial winding 
number. We also give a recursion formula for these Laurent polynomials and 
generalise the trace formulas for the groups generated by complex /^-reflections. 

We apply these formulas to prove some discreteness and some non-discre- 
teness results for complex hyperbolic triangle groups. 



1. Introduction 

We study representations of real hyperbolic triangle groups, i.e. groups generated 
by reflections in the sides of triangles in i/g, in the holomorphic isometry group 
PU(2, 1) of the complex hyperbolic plane H^. 

We use the following terminology: A complex hyperbolic triangle is a triple 
(Ci, C2, C3) of complex geodesies in H^. If the complex geodesies Ck-i and Ck+i 
meet at the angle Tr/pk we call the triangle (Ci, C2, C3) a (pi, p2, P3)-triangle. If the 
complex geodesies Ck-i and Ck+i are ultra-parallel (i.e. their closures in H^LidH^ 
do not intersect) with distance £k we call the triangle (Ci,C2,C3) a [-^1,^2,^3]- 
triangle. A complex geodesic (or complex slice) in is the fixed point set of a 
complex reflection. A complex reflection is an element of PU(2, 1) conjugate to the 
map 

[zi : Z2 : 2:3] [zi : ~Z2 : -2:3] • 
For more details on complex reflections and complex and real slices see section 2. 

We call a subgroup of PU(2, 1) generated by complex reflections ik in the sides Cfc 
of a complex hyperbolic (pi,p2,P3)-triangle (Ci, C2, C3) a {pi, p2, P3)-triangle group. 
We call a representation of the group 

r(pi,P2,P3) = (7i;72,73 I 7fe = ilk-iJk+iT" = 1 for aU k e {1,2,3}), 

where 7fc-i-3 — 7fc, and the relation {'-/k-ijk+i)^'' = 1 is to omit for pk — 00, into the 
group PU(2, 1) given by taking the generators 7^ of r(pi,p2,P3) to the generators 
tfc of a (pi,p2,P3)-triangle group a {pi,p2,P3)-representation. In the ultra-parallel 
case we define similarly [£i, £2, (al-triangle groups and [£i, £2, (sl-representations. 
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For fixed {pi,p2,P3) resp. (i'i,i?2,^3) the space of complex liyperbolic triangle 
groups is of real dimension one. There is a canonic;al path pt of representations. 
The starting point po for the path is the case when the triangle lies in a real slice. 
Real slices are the second type of totally geodesic subspaces in . 

For (pi,p2,P3) -triangle groups, according to Richard Schwartz [Sch02], the con- 
jectural picture is as follows: 

Conjecture. We assume pi ^ P2 ^ Pa- Define wa = t,3i'2t'3i'i and wb = i'it2i'3- 

A (pi,p2,P3) -representation is a discrete embedding if and only if neither wa nor 
Wb is elliptic. The set of the corresponding parameter values is a closed symmetric 
interval. 

If the element wa becomes elliptic before wb, we say that the triple {pi,P2,P3) 
is of type A. else we say that the triple is of type B. 

The triple {pi,p2,P3) is of type A if pi < 10 and of type B if pi > 13. 

// the triple {pi,P2,P3) is of type A then there is a countable collection of pa- 
rameters, for which the {pi,p2,P3) -representation is infinite and discrete but not 
injective. If the triple {pi,p2,P3) is of type B then there are no such discrete but 

not injective {pi,p2,P3)-representations. 

R. Schwartz proved this conjecture for (oo, oo, oo)-groups in [SchOlb], and for 
(Pi)P2,P3)-groups with pi, p2, Ps sufficiently large in [Sch03b]. For sufficiently 
large pi, P2, Pa the triple (pi,P2,P3) is of type B. 

The case of (p,p, oo)-triangle groups was studied in [WGOO] by Justin Wyss- 
Gallifent. It turns out that the triple {p,p, oo) is of type A for p < 13 and of type B 
for p ^ 14. He also described some of the discrete but not injective (4, 4, oo)- 

represcntations. 

In general, to prove that a certain (pi,p2,P3)-representation is discrete and injec- 
tive, it is necessary and sufficient to show that the images of elements of r(pi,p2,P3) 
of infinite order arc not regular elliptic. Here the traces come in, namely, an cle- 
ment of SU(2, 1) is not regular elliptic if and only if the value of certain discriminant 
function (introduced by W. Goldman) on the trace of the element is not negative. 

The main results of this paper are formulas for traces of elements in complex hy- 
perbolic triangle groups, a combinatorial trace formula (theorem 4) and a recursive 
trace formula (theorem 7) as well as their applications. These formulas generalise 
the results of Hanna Sandler [San95] on ideal triangle groups. 

We parameterise the one-dimensional space of (pi,p2,P3)-triangle groups by an 
invariant a of triangles in the complex hyperbolic plane. The combinatorial trace 
formula express the trace of an element of the group as a Laurent polynomial in e'" 
with coefficients independent of a and computable using a certain combinatorial 
winding number. Then we give a recursion formula for these Laurent polynomi- 
als. We also generalise the trace formulas for the groups generated by complex 
/i-reflections. 

We apply the formulas to prove some discreteness and some non-discreteness 
results for complex hyperbolic triangle groups. 

For instance, we compute the parameter value tA such that for \t\ > tA the 
corresponding (pi,p2,P3)-representations are not discrete embeddings because the 
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element wa is regular elliptic (proposition 12 and corollary 13). Furthermore, for 
a certain subfamily of triangle groups we can partially confirm the conjecture of 
Schwartz about type A and B. We find a sufficient condition for a triple {pi,P2,P3) 
to be of type B. This condition implies for example that triples {p,p, oo) withp ^ 14 
and triples (p, 2p, 2p) with p > 12 are of type B (proposition 16). We also obtain 
similar results for ultra-parallel triangle groups. 

The paper is organised as follows: In section 2 we recall the basic notions of 
complex hyperbolic geometry, specially for the complex hyperbolic plane H^. In 
sections 3 and 4 we define angular invariant a, which classifies complex hyperbolic 
triangles with fixed angles up to isometry, and we compare this invariant with 
invariants defined for some special cases by U. Brehm, J. Hakim and H. Sandler. 
Section 5 contains the description of certain combinatorial functions of words, first 
of all the winding number. 

After that we are prepared to state and to prove in sections 6 and 7 our main 
results, the combinatorial trace formula (thm. 4) and the recursive trace formula 
(thm. 7). In section 8 we discuss as the first application of the recursive trace 
formula the resulting formulas for the traces of short words in triangle groups. 
We prove in section 9 some properties of the traces of elements in a triangle group, 
which are also used later in section 13. In section 10 wo describe the generalisations 
of our trace formulas for groups generated by /^-reflections (for the definition see 
section 2). 

Sections 11-13 contain applications of the trace formulas. In section 11 we discuss 
necessary conditions for a triangle group representation to be a discrete embedding. 
Section 12 deals with a subfamily of (pi,p2,P3)-triangle groups with the property 
^ = ^ + ^ resp. of [£i,£2,^3]-triangle groups with the property £^ — l\ -\- £2- The 
triangle groups in this subfamily seem to share a lot of properties of the ideal triangle 
groups. For this subfamily we can prove more precise statements about type A 
and B. In section 13 we discuss some arithmetic properties of the traces for certain 
(finite) family of signatures and parameter values of triangle groups. Finally, in 
section 14 we summarise some of the known results on complex hyperbolic triangle 
groups and describe them in terms of our parameter ri, r2, ra, and a (for notation 
compare section 3). 

Part of this work was done during an enjoyable stay at the University of Mary- 
land. I am grateful to William Goldman and Richard Schwartz for their kind 
invitation and for helpful discussions and to the University of Maryland for its 
hospitality. I would like to thank Werner Ballmann, Egbert Brieskorn, Martin Der- 
aux, Ilya Dogolazky, Tadeusz Januszkiewicz, John Parker, Blake Pelzer, and Gregor 
Weingart for useful conversations related to this work. 

2. Complex Hyperbolic Plane 

In this section we recall some basic notions of complex hyperbolic geometry. The 
general references on complex hyperbolic geometry are [Gol99] , [Par] , and [BH99] . 

Complex Hyperbolic Plane: Let C^'^ denote the vector space C'^ equipped 
with the Hermitian form 



{Z, W) = ZiWi + Z2W2 - Z3W3 
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of signature (2,1). We call a vector z G C^'^ negative, null, or positive, according 
as {z, z) is negative, zero, or positive. Let P(C^'^) denote the projectivisation 
ofC2'^-{0}. We denote the image of z S C^'"'^ under the projectivisation map by \z\ . 
We also write \_z\ = \z\ : zi : 23] for z = (z\,zi, 2:3) G C ' . The complex hyperbolic 
plane is the projectivisation of the set of negative vectors in C^'^. Its ideal 
boundary dH^ is defined as the projectivisation of the set of null vectors in C^'^. 
The complex hyperbolic plane is a Kahler manifold of constant holomorphic 
sectional curvature. The holomorphic isometry group of is the projectivisation 
PU(2, 1) of the group SU(2, 1) of complex linear transformations, which preserve 
the Hcrmitian form. 

The Hermitian cross product: M : C^'^ x C^'^ C^'^ is defined by 

(-1 0\ / 2:3^2 - Z2W3 \ 

—1 (z X w) = 2:1^3 — Z3W1 j . 
1/ \Z1W2 — Z2W1 J 

The Hermitian cross product of two vectors yields a vector perpendicular to both 
of them 

{z ^w,z) = {z^ w, w) = 0. 

Furthermore we have 

{a^c,bMc) = {a,c){c,b) - {a,b){c,c), 

in particular 

(aH6,aH6) = \{a,b)\^ - {a,a){b, b). 

Totally Geodesic Submanifolds: There are two kinds of totally geodesic 
submanifolds of dimension 2 in , complex slices (or complex geodesies) and real 
slices (or totally real totally geodesies subspaces). Complex geodesies are obtained 
by projectivisation of 2-dimensional complex subspaces of C^'^. Given any two 
points in H^, there is a unique complex geodesic containing them. Any positive 
vector c e C^'^ determines a 2-dimensional complex subspace 

{zeC^'^ I {c,z) = Q} 

and a complex geodesic, which is the projectivisation of this subspace. The vector 
c is called a polar vector of the complex geodesic. A polar vector can be normalised 
to (c, c) = 1. Conversely, any complex geodesic is represented by a polar vector. 
A typical example is the complex slice {[z : : 1] € H^} with polar vector c = 
(0, 1,0)"^. Any complex slice is isometric to this one. 

We now describe possible configurations of two complex geodesies (compare 
proposition 6.8 in [Par]). For two complex geodesies one of the following situa- 
tions occur: 

• they coincide, 

• they intersect in a single point in iJ^, 

• their closures in U dH^ intersect in a single point in dH^ , 

• their closures in H"^ U dH'^ are disjoint. 

Let Ci and C2 be two complex geodesies with normalised polar vectors ci and 
C2 respectively. The complex geodesies Ci and C2 intersect in H'^ if and only if 
|(ci,C2)| < 1, in which case |(ci,C2)| = cosZ(Ci,C2), where Z(Ci,C2) is the angle 
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of intersection between Ci and C2, and ci C2 is a negative vector, corresponding 
to the intersection point. The angle of intersection can be defined as 

^(^1,^2) =min{Z(AiCi,A2C2) | Ai,A2 eM}, 

where the angles on the right hand side arc measured in the underlying real vector 
space of C^'^. The angle satisfies Z{Ci,C2) e [0,7r/2]. The complex geodesies Ci 
andC2 intersect in 97?^ ifandonlyif |(ci,C2)| = 1, in this case ciKlc2 is a null vector, 
corresponding to the intersection point. The closures of the complex geodesies Ci 
and C2 are disjoint if and only if |(ci,C2)| > 1, in which case |(ci,C2)| = cosh(£/2), 
where £ is the distance between Ci and C2. 

Any real slice is isometric to {[z : w : 1] G \ z,w & M}. Real slices are fixed 
point sets of real reflections, i.e. antiholomorphic isometries conjugate to the map 

[z : w : 1\ ^ [z : w : 1]. 

Complex Reflections: Given a complex geodesic C, there is a unique isome- 
try be in PU(2, 1) of order 2, whose fixed point set equals C. We call this isometry 
the complex reflection in C (or the inversion on C). The complex refiection in C 
is represented by an element lc € SU(2, 1) that is given by 

ic{z) = -z + 2^^c, 
{c,c) 

where c is a polar vector of C. 

Complex /i-Reflections: For a unit complex number fj, a complex /x-reflection 
is an element of PU(2, 1) conjugate to the map 

[zi : Z2 ■■ Z3] [^zi : Z2 : 23]. 

Its fixed point set is a complex geodesic and the /z-refiection rotates around this 
complex geodesic by the angle arg(/i). For n = —1 we get the usual complex 
reflections described above. The complex /i-refiection in a complex geodesic C with 
a polar vector c is given by 

Classification of isometries: An isometry (p of is called elliptic if it has 
a fixed point in H^. It is called hyperbolic (or loxodromic) if its displacement 
dip = inf{d{x, Lp{x)) | x € H^} is positive. Here d is the inner metric on i/^. 
A hyperbolic isometry has two fixed points that lie in dH^. An isometry, which 
is neither elliptic nor hyperbolic, is called parabolic. It has one fixed point that 
lies in dH^. So far this was the usual classification of isometrics of non-positive 
curved spaces. In the case of the complex hyperbolic space we can refine this 
classification. An elliptic element is called regular elliptic if all its eigenvalues are 
distinct. A parabolic element, which can be written as an element of U(2, 1) with 
only eigenvalue 1, is called unipotent, otherwise the parabolic element is called 
ellipto-parabolic (or skew-parabolic). 

Using the discriminant function 

p{z) = \z\^ - 8Re(.2^) + 18\zf - 27 
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we can classify isometrics of the complex hyperbolic plane by the traces of the 
corresponding matrices: An isomctry A G SU(2, 1) is regular elliptic iff p(trace A) < 
and hyperbolic iff p(traceA) > 0. If p{tTa.ceA) = there are three cases. If 
(trace A)'^ — 27 then A is unipotent. Otherwise, A is either a complex reflection in 
a complex geodesic or a complex reflection about a point, or A is ellipto-parabolic. 
The proof can be found in [Gol99], Theorem 6.2.4. Note that for real z the function p 
factors into p{z) = {z + l){z — 3)^. This means for A e SU(2, 1) whose trace is real, 
that A is regular elliptic iff trace ^4 e (—1, 3) and hyperbolic iff traced ^ [—1, 3]. 



3. Complex Hyperbolic Triangles and a-lNVARiANT 

In this section we describe a parameterisation of the space of (complex hyper- 
bolic) triangles in H^, i.e. of triples (Ci, C2, C3) of complex geodesies, by means of 
an invariant a. Let Cfc be the normalised polar vector of the complex geodesic Cfe. 
Let Tfe = \{ck-i,Ck+i)\- If the complex geodesies Ck-i and Ck+i meet at the angle 
(fik, then Tfe = cosi^fe. If the complex geodesies Ck-i and Ck+i are ultra-parallel 
with distance £k, then = cosh(^/;/2). We define the angular invariant a of the 
triangle (Ci, C2, C3) by 

a = arg ^]J(cfc-i, Cfe+i) j . 

The angular invariant is obviously invariant under isometrics of H^. The complex 
reflection ik = the complex geodesic Cfc is defined by 

'-fe(^) = -z + 2{z,Ck) -Ck- 

(Note that because of the property {ck, Ck) = 1 we can simplify the formula for the 
reflection.) 

Remark. If the complex geodesies C'k-i and Ck+i or their closures intersect, then 
the vector Vk = Ck-i^Ck+i is a negative resp. null vector, which corresponds to the 
intersection point, i.e. to a vertex of the complex hyperbolic triangle. The points 
[vk] can be also meaningfully interpreted in the case of ultra-parallel geodesies Ck-i 
and Cfc+i. 

We call a complex hyperbolic triangle (Ci,C2,C3) a ((/^i, ^2, </'3)-triangle if the 
complex geodesies Ck-i and Ck+i meet at the angle ifk- 

Proposition 1. A {(pi,(p2,<f 3) -triangle in is determined uniquely up to isom- 
etry by the triple (vi, V2, Vs) and the angular invariant 

a = arg ^ (cfc_i, Cfe+i)^ . 

For any a G [0, 27r] there exists a {(pi, ip2, 'Ps) -triangle in with angular invari- 
ant a if and only if 

r? + rn +r^ — 1 

cos a < — — , 

2rir2r3 

where rk = cos ipk ■ 
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Proof. We can normalise so that 

C2 = \ 5 \ , and C3 




with ^ e C, ^, /3, 7, ^ e M and ^, 7, 5 > 0. We compute 

(C1,C3)=^, (C2,Ci) =70 + 5^, (C3,C2)=^. 

The conditions |(cfe_i, c/;+i)| = ru and (cfe.Cfc) = 1 imply 
d = ri, 7 = si, 

C = r2, 1^1 >S2, /3=(N2 + ri-l)V2, 
-2= (rse"*" - rir2)srS 



where rfc = cos(<^fc) and Sfc = sm{ipk). The inequality |2;| > S2 finally implies 
|r3e*" — rir2| > siS2- Computation shows that this inequality is equivalent to 
cosa < (r^ + r| + r| — I)(2rir2r3)~-^. □ 

Remark. In the case of angular invariant a = tt all the vertices [vi], [v2\, and [v^] of 
the triangle lie in one real slice. The corresponding triangle group representation 
stabilises this real slice. The angular invariant a with 

r? + To + ri - 1 

cos a = 

2rir2r3 

corresponds to the case that all vertices [vi], [V2], and [v^] of the triangle coin- 
cide. The groups with angular invariant a and 27r — a are conjugate via an anti- 
holomorphic isometry of . For this reason we can restrict ourselves to the cases 
of angular invariant a G (0,7r]. 



Similar statements can be proved also in the cases when some of the complex 
geodesies are ultra-parallel. For example, for an ultra-parallel 2, ^3] -triangle 

(Ci, C2, C3), i.e. in the case that the complex geodesies Ck-i and Ck+i are ultra- 
parallel with distance /'j., we obtain by similar reasoning the following result: 

Proposition 2. An ultra-parallel [£i, £2, f-sl-triangle in is determined uniquely 
up to isometry by the triple (i?i,^2,^3) and the angular invariant 

a = arg ^ JJ(cfe-i, Cfe+i) j . 

For any a € [0,2?:] there exists an ultra-parallel [£i,i2,£3]-triangle in with 
angular invariant a if and only if 

r? + To + r? — 1 

cos a < , 

2rir2r3 

where rk = cosh(^/;/2). 
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Examples of explicit parameterisations: In the case = oo we consider 
for a e [— TT, tt] the complex geodesies with normalised polar vectors 





















{-J 








where zi = cos(7r/pi)e~'"/^ and Z2 = cos(7r/p2)e*"/^. Computing 

(ci, C3) = r2e^"/2, (C2, Cl) = 1, (C3, C2) = rie'«/2 

we see that i-Ck defines a {pi,P2, oo)-representation with a-invariant equal to 
a. For the sake of completeness we also compute the vertices of the triangle 

= [-rie^«/2 : : 1], [V2] = [-r2e-^«/2 : : 1], [vs] = [0:1: -1]. 

A special case of this parainetcirisation is the paramctcrisation of {p. p, oc)-groups 
in [WGOO] . Another example of explicit parameterisation is the parameterisation 
of (4, 4, 4)-groups in [Sch03a]. 

4. Connection of the Angular Invariant a and Other Invariants 

In this section we compare our parameterisation of the space of complex hyper- 
bolic triangles with other parameterisations by Ulrich Brehm [Bre90] and Jeffrey 
Hakim and Hanna Sandler [HSOO]. 

Let {Ci,C2,Cs) be a complex hyperbolic triangle. Let Ck be the normalised 
polar vector of the complex geodesic Ck- Assume that for k = 1,2,3 the complex 
geodesies Ck-i and Cfe+i or their closures intersect. Let Vk = Ck-i ^ Ck+i- 

The Cartan angular invariant of three points [vi], [V2], and [vs] in dH^ was 
defined by E. Cartan in [Car32] as follows: 

^([wi], [V2], [wa]) = a.rgi-(vi,V2){v2,va){v3,vi)). 

To compare the Cartan angular invariant and our invariant a we need the following 
proposition: 

Proposition 3. Let {ck-i,Ck+i) = rfee'^''. Then we have {vk,Vk) = — 1, 
{vk-i,Vk+i) = e'^'=(rfe - rfe_irfe+ie-'"), and 

33 3 
JJ(?;fe,-!;fe+i) = {vk-i,Vk+i) = e~'" • - rfe_ir/c+ie'"). 

fe=i fc=i fe=i 

Proof. The proof of this proposition, based on the fact that [vk] = [ck-i ^ Ck+i] 
and the formula for (a Kl c, 6 Kl c), is straightforward. □ 

Using the last proposition we obtain the following formula for the Cartan angular 
invariant 

A = arg(-e-'"(l - e'^f) = mod27r 

because of arg(l — e'") = {a — 7r)/2. 

Ulrich Brehm defined in [Bre90] the shape invariant of three points [vi], [V2], 
and [vs] in iJ^ ^ 

'(z;i,i;2)(?^2,i^3)('y3,vi)' 



o-([wi], [v2\, [V3]) = Re 



(wi,t;i)(W2,t;2)('!^3,W3) 
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The shape invariant a is invariant under PU(2, 1), and the triangle [vi], [V2], [fa] 
is determined uniquely up to isometry by its three angles together with the shape 
invariant. Using proposition 3 we obtain the following formula that describes the 
relation between the Brehm's shape invariant a and our invariant a: 

_ ■'"i^i'"! cos 2a — rir2r3(r^ + + rf + 1) cos a + (r^r^ + r|r| + r§r^) 

(l-rf)(l-ri)(l-ri) " 

In [HSOO] Jeffrey Hakim and Hanna Sandler defined such angular invariants also in 
the cases of triangles with one or two ideal vertices: 

{vz,Vi){vi,V2) 



{v3,V2){vi,Vi) 

Using proposition 3 we compute 

{r2 — rie*")(l — rir2e*" 



T] = e 



(n - r2e'«)(rf - 1) 



5. Winding Numbers and Other Combinatorial Functions 

In this section we introduce some combinatorial functions of words, which will 
appear in the combinatorial trace formula. 

We first fix some notation. Elements of the (pi,p2,P3)-triangle group, or the 
abstract group T{pi,p2,P3), may be naturally identified with finite sequences of 
numbers 1, 2, and 3. We refer to such sequences as words. The sequence a = 
(fli, . . . , a„) corresponds to the element ia '■= t-a^ ■ ■ ■ ia^- We shall simplify notation 
where it is convenient, for example we write (123) instead of (1,2,3) and (,123 instead 
'•(123) or (-(1,2,3)- The empty sequence corresponds to the identity element denoted 
by i0. There is a natural definition of the powers (ai, . . . , a„)"' for w £ "Z, which 
is consistent with taking powers of the corresponding elements in the group, for 
example (123)^ = (123123) and (123)"^ = (321). We denote by Ta the trace of the 
element ta- 

We now introduce the winding number. Let x denote the unique nontrivial 
character modulo 3, i.e. if a is any integer then x(a) is the unique element of the 
set of integers {—1,0, 1}, which is congruent to a modulo 3. Define 



1 

w{ai, . . . , a„) = - • ^ xio-m+i - am), 



3 

m=l 



where a„+i := ai. 



Remark. The integer w{ai, .. .,an) can be interpreted as the winding number of 
the loop ai —> a2 ctn ai along the vertices of the (1, 2, 3)-triangle (see 

figure 1). 

We are interested in traces of elements. For Ai, . . . , € SU(2, 1) we have 

trace(Ai • • • Am) = trace(A„ ■ Ai--- Am-i)- 

Since the trace of a product docs not change under cyclic permutation of the factors, 
we consider cyclic words instead of usual (linear) words. A cyclic word is the orbit 
of a linear word under cyclic permutations. 



10 



ANNA PRATOUSSEVITCH 



2 



1 3 

Figure 1: Winding Number and (1, 2, 3)-Triangle 

Dealing with winding numbers we often use the following consideration: We start 
with the following operations on cyclic words: reduction 

(■ ■ ■ jkjk, - ■ ■) > (• • • , fc, • • • ) 

and straightening 

(• • • ,k,l,k,---) > (• • • , /c, • • • ). 

These operations do not change the winding number, and any cyclic word a can be 
transformed into the cyclic word (123)"'^"' using these operations. 

We now introduce combinatorial functions Uk that will be used in the combina- 
torial trace formula. 

Let 

Ufe(ai, . . . , a„) = |{m e {1, . . . , n} | {am, (Xm+i} = {k-l,k + 1}}\, 
where a„+i = ai. Using the function ipk given by tpk{0', 6) = 1 iff 

{a,b} = {k-l,k + l} 
and tpk{0; b) = otherwise we can describe Uk as 

n 

Ufe(ai, . . . ,a„) = ^ ipk{am, dm+i), 

where again a„+i = ai. 

6. Combinatorial Trace Formula 

The first main result of this paper is the following combinatorial formula for 
traces of elements in complex hyperbolic triangle group. 

We first fix some notation. For a word a = (ai,...,a„) and a subset S of 

the set [n] := {1, . . . ,n} we denote by Uk{S) resp. w{S) the values of Uk resp. w 
for the corresponding subsequences of a, for example Uk{S) = u/c(aii, . . . ,ai^) for 
S = {ii, im} with l^ii < ■■■ <im^n. 

Theorem 4. Let a= (ai, . . . , a„) be a cyclic word, ta = • • • ia„ the correspond- 
ing element of the {pi,P2, Pa) -group and Ta the trace of ta- Then 

V Sc[n] J 
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Equivalently, Ta = (—1)" • (2 +Pa{e^")), where Pa{z) = '^^QwZ^ is the Fourier 
polynomial with coefficients 

q^= (-2)1^1 -rr^^Vr^^Vr^^^ 

SC[n] 

Proof. We write the complex reflections as ik = — id+2cfecj^, where c* is the dual 
vector to a vector c e C^'^, i.e. c*{z) = {z,c). To get a formula for traces we 
expand 

= (-id+2c„,c:j...(-id+2c„„c:j 

S={ii,...,i^}C[n] 



Let S = {ii, . . . ,ijn} be a subset of [n]. For ease of notation let bj = at^ and 
b = (fei, . . . , bm)- We have 

and trace(c(,i c^^ ) = cj^ {cb^ ) implies 

trace(c6i c^^ • • • c^^ ) = c^^ (cb^ ) • • • C6„_, {cb^ ) • c^^ (cft^ ) 

= {Cb2,Cbi) ■■■{Cbm,Cb^-i) ■ {Cbi,CbJ. 

We claim that 

(C6.,C6,) • • • (C6„,c,_,) • {cb,,cbj = rr^'^r^^r^'^ • e^--w. 

It is clear that the absolute value of the complex number on the left hand side of the 
equation is equal to r^^^^V2^^''V3'*^''\ The statement about the argument of this 
complex number is clear for the words of the form (123)"" for some w G li. Any other 
cyclic word b can be transformed into the word (123)"'^''^ using the winding number 
preserving operations of reduction and straightening as described in section 5. It 
remains to note that this operations do not change the argument of the complex 
number on the left hand side of the equation. So we have proven the formula 

for S = {ii, . . . , im} C [n]. (Note that this formula does not hold for 5 = 0, so we 
have to handle this case separately.) Using this formula we finally obtain 

Ta = (-1)" • ( 2 + E (-2)1^1 • r^^(s)r^-(s)^u,is) _ ^,„^(s) j _ □ 

Remark. In the ideal triangle group case, i.e. for pi = p2 = p^ = 00, it follows from 
the Binomial Theorem 

Eft= E (-2)'^' =(1 + (-2))" = (-!)"• 

k Sc[n] 
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7. Recursive Trace Formula 

In explicit computations we rather use the foUowing recursive trace formula, 
which can be derived from the combinatorial trace formula of the last section. We 
first state some properties of the winding number and of the combinatorial function 
Mfc. They follow from routine induction arguments. 

Proposition 5. We note that 

w{ai, . . . , o„) = w{ai, am.) + w{am, • . • , a„) + w{ai,am, dm+i, a™) 
for TO = 2, . . . , n — 1. Especially 

w{ai, . . . , a„, a„+i) = w{ai, . . . , a„) + w(ai, a„, a„+i), 

and 

w{bl, .. .,bm, an-2, an-l,an) = w{bl, . ..,bm, an-2, an) + w{an-2, (Xn-l, On)- 

Proposition 6. We note that 

Wfc(oi, . . . , o„, a„+i) = Mfc(ai, . . . , a„) + Vk{an, a„+i, ai) 

and 

Uk{bl, ■ ■ ■ -ibk, an-2, On-l) fl^n) = Wfe(5i, . . . , 6fe, an-2, On) + Vk{an-2, «n-l) ^n)) 

w/iere 

Ufc (a, 6, c) = Vfc (a, ?>) + V"/: c) - tjjk (a, c) . 
Define deletion operators d, 6', and 6": For a word a — (ai, . . . , a„) let 

S{a) = (ai, . . . ,a„_3,a„_2,a„-i), 
(5'(a) = (ai, . . . , a„_3, a„_2, o„), 
J"{a) = (ai, . . . , a„_3, a„_i, a„). 

Theorem 7. Lei a = (oi, . . . , a„) 6e a worrf o/ length n > 3 anrf To t/ie irace o/ t/ie 
corresponding element La- Then 

Ta = -{na + TS"a + TsS"a) + /3(T5'a + + TsiS"a + TS5'S"a), 

where 

with T = {an-2,an-\,an). 
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Proof. The proof is along the hnes of the proof of Theorem 3 in [San95]. We 
consider 



i„ : = (-1)"t„ - 2 



r, r2 e 



Sc[n] 

ta'=ta- tSa = (~l)"'(Ta + Tga) 

= ^ (^_2yS\ .j.uiiS)^U2(S)^U3{S)^iaw{S)^ 

neSC[n] 

t'a • = *a - *5'a = (-l)"(T'a + T^a + Ts'a + m'a) 

= ^ (•_2)1'S| . y,«i(S)^«2(S)^«3(S)giat«(S)^ 

n-l,neSC[n] 

./// ,// _ ,1/ 

"-a • ^ '-a ''(5" a 

= ( — l)"(ro + T5a + Tg'a + Ta^'a + T5"a + TsS"a + TS'S"a + TSS'5"a) 
^ ^ (_2)I'S| . ^«i('S)^W2(S)^M3(S')gia«;(S)_ 

ra-2,ra-l,rae5'C[n] 

For S C [n - 3] let 5 := 5 U {n - 2, n - 1, n} and S' := S U {n - 2, n}. By 
propositions 5 and 6 it holds for S C [n — 3] 

w{S)=w{S')+wiT) and Uk{S) = Uk{S') + Vk{T). 

This implies 

= ^ (_2)|5'| . ^^i(5')^«2(S)^M3(S)gic«t«(S) 

n-2,n-l,neSC[n] 
= ^ (_2)|S|+3 . ^«i(S)^^2(S)^U3(S)^iau,(S) 

Sc[n-3] 

= _2 . r^i(^)r^2(T)^^3(T)gja^(T) . ^ (-_2)|S'|+2 . ^«i(S")^^2(S')^«3(S')gic,«,(S') 

Sc[n-3] 

Prom this equation and the definition of t'^' we obtain 

t'a = 4'"o + t'a = *5"a " (/^ + 1) ' *5'a- 

On the other hand t" can be expressed in terms of the traces as 

(-l)"(Ta + TSa + TS'a + TsS'a) 

= (-l)"~"^(T5"a + TS5"a + TS'5"a + T55'5"a) 

~ (/3 + 1) • (-l)"~"^(T5'a + + r^'^'a + m'j'a). 
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This implies using 5' 5' = S'5" 

Ta = — (r5"a + TSS" a + TS'S"a + TsS'S"a + Tga + TS' a + TSd'a) 
+ (/3 + 1) • {TS'a + T55'a + T^'^"a + T^S' S" a) 
= -{Wa + TSS"a + TSa) + /? ' {tS' a + ^'o + TS'5"a + T55'5"a)- ^ 



In order to use this formula to compute traces recursively, it is necessary to know 
the traces of all elements associated to words of length 0, 1, and 2: 

T0 =3, Tk — —1, 

T"fe-i,fe+i = Tk+i^k-1 = 4r^ - 1- 



8. Examples 

We compute the traces for some special elements of complex hyperbolic triangle 
groups, in particular for wa = '-1323 and Wb = (-123- 

We have 

n23 = -(t12 + T23 + T2) + ■ (ti3 + Ti + T3 + Tg,), 

where 

P = 2^-i(i.2,3)^..(i,2,3)^.3(i,2,3)gia^(i,2,3) _ j ^ 2nr^'rse'" - 1. 

Hence 

T123 = -((4r| - 1) + (4r? - 1) + (-1)) + {2nr^\se''' - 1) • {{irl - 1) + 1) 
= Snrarse'" - (4(rf +r^+ r^) - 3). 



Furthermore we have 

T2321 = — (t232 + T22I + T22) + P ■ {t231 + T23 + T21 + T2), 

where 

/3 = 2,.-i(3.2,i)^^2(3,2,i)^.3(3,2,i)^,„^(3,2,i) _ j ^ anra-Vse"- - 1. 

It holds i23i = ti(ti23)'^r^ ^'^d therefore T231 = ti23- Because of r232 +T221 +T22 
T3+T1+T2S = 1 and 

T231 + 7'23 + T21 + T2 

= (8rir2r3e'« - (4r? + 4r^ + 4r^ - 3)) + (4r? - 1) + (4r^ - 1) + (-1) 
= 8rir2r3e*" - 4r^ = 4r2(2rir3e'" - r2) 
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we obtain 

T2321 = -1 + (2rir2-V3e-*" - 1) • Ar2{2r^r^e"^ - ra) 
= 4(2rir3e-*" - r2)(2rir3e*" - r^) - I 
= 4-|2rir3e'"-r2|^-l 
= {l&r\r\ + — 1) — 16rir2r3 cos a. 

Similarly, for any k we get 

<yk '■= Tk^k-l.k,k+l = Tk^k+l,k,k~l 

= 4- |2rfe_irfe+ie*" - rfe|2-l 

= {16rl_irl_^_i + 4rl — 1) — 16rir2r3 cos a. 

9. Some Properties of the Traces 

In this section we study some properties of our trace formulas. Lemma 8 will be 
also used later in section 13. 

Lemma 8. The coefficients in the combinatorial trace formula satisfy 

g«,€(8rir2r3)l'"I.Z[4r2,4r2,4r2]. 
Proof. The coefficient is a sum of the terms of the form 

_l_2|6| . . ^^2(6) , j,«3(fc) 

for some word h of length |fe| and winding number w{h) = w. For the words 
b = (123)^ with u; e Z we have 

21"! . r^^(b) ^u,(b) ^u,ib) ^ 231-1 . ^I-I . . ^I-I = (8rir2r3)l'"l. 

Any other word b can be transformed into the word (123)"'*-''-' using the winding 
number preserving operations described in section 5. But the number 



(b) 



is divided by 2 under reduction and by 4r| for some k € {1,2,3} under straighten- 
ing. Altogether this implies 

±2l^l • rT^'^ ■ rT^'^ ■ r^^'^ e (8rir2r3)l"'('')l • Z[4r^4r^,4r^] 
and hence q^ e (8rir2r3)l'"l • Z[4rf , 4r|, 4r|]. □ 

Theorem 9. For any cyclic word a of length n there is a polynomial Sa{z) in 

Z[4r^, 4r|, 4r|][z, z] of degree at most n/2 in 4r^ and at most n/3 in z and z, 
independent of the angular invariant a, such that Ta = Sa(r), where t = ri23. 

Remark. In particular Sa = Bk for a = (123)'^, where polynomials Bk are the 
analogues of the Chebyshev polynomials for SU(2, 1) as defined by Hanna Sandler 
in [San95]. 
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Proof. Let R :— Z[4ri, 4r2, 4r|]. The tracer = ri23 is equal tor = (8rir2r3)-e*"— c, 
where c = Arl + 4r2 + 4r3 — 3 e -R. Therefore 

e'««' = (Snrsrs)-!"'! • ([r or f] + G (Snrars)"'"'! • R[r,f]. 

On the other hand the coefficients in the combinatorial trace formula satisfy 
q^, € (Snrarg)!'"! • R by lemma 8, hence ■ e*"*" e i?[r, f]. □ 

10. Groups Generated by Complex /^-Reflections 

In this section we generalise the trace formulas for the case of groups generated 
by complex /i-reflections. 

Let {Ci,C2,C3) be a complex hyperbolic triangle. Let Ck be the normalised 
polar vector of the complex geodesic Ck, and let rk = \{ck-i,Ck+i)\- The angular 
invariant a of the triangle (Ci, C2, C3) is 

a = arg ^]J(cfe_i,Cfe+i)^ • 

Let /xi, fi2, and /X3 be unit complex numbers. The complex /i/;-reflection tk = t'ck 
in the complex geodesic Cfc is given by 

Lkiz) = Z + (/Xfe - 1) • {Z, Ck) ■ Ck- 

We consider the subgroup generated by the reflections li, L2, and 13. The formulas 
for traces can be generalised for this case. 

Let 

nfe(oi,...,a„) = |{me {!,..., n} | a„ = fc}|. 

For a word a = (ai, . . . , o„) and a subset S of the set [n] :={!,..., n} we denote 
by nk{S), Uk{S), resp. w{S) the values of n^, Uk, resp. u; for the corresponding 
subsequences of a, for example Uk{S) = Uk{ai^ , . . . , Oj^) for 5 = {ii, . . . , im} with 
1 < n < • • • < im ^ n. 

Theorem 10. Let a = (ai,...,a„) be a word, La = tai ' ' ' '^On corresponding 
element of the group generated by the three reflections ik = f-c'k '^^'^ '''^ trace 
of La- Then 

Sc[n] \k=l J \k=l J 

Proof. We write the complex /x^-reflection as Lk = id+(/ife — l)cfeC^, where c* is the 
dual vector to a vector c € C^'^, i.e. c*{z) = {z,c). In order to get a formula for 

traces we expand 

~ ^ai * * * 

= (id+(/Xai - l)CaiC*J---(id+(/Za„ - l)Ca„<^) 
5={ii,...,i„}C[n] \fe=l / 



Now we proceed as in the proof of theorem 4. 



□ 
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The generalisations of other statements based on the combinatorial trace formula 
for the case of the groups generated by /^-reflections are straightforward, for instance 
the generalisation of the recursive trace formula and of lemma 8. 

As an example, wc consider Mostow's non- arithmetic complex hyperbolic lat- 
tices [Mos80]. Let us denote by T{p,p) a complex hyperbolic triangle groups gen- 
erated by complex /i-reflections of the same finite order p e {3, 4, 5} 

/i := /ii = /i2 = Ms = e p 
in the sides of an equiangular triangle with 

1 

r := n = r2 = rs 



2sin^ 

p 



and the value of the angular invariant a of the form 

27r TT TT 

a=— + ---. 
p p 2 

Proposition 11. The traces of elements in the group T{p,p) satisfy 

Ta e Q[e p ,e p \. 
Proof. Generalising lemma 8 we obtain that the coefficients 



= E n(M-ir^^M- n 

scH \fe=i / \fe=i 



in the combinatorial trace formula satisfy 

q^&{{^l-l)rf\^\■■L[^i, (M-i)V]. 

2-Ki H 

For fj, = e p and r = ^ holds 

p 

{n — l)r = i ■ e~ , 



hence we obtain 



and therefore 



Ta e Q[e^,ie'?e°']. 



Finally 



implies, that the traces of elements in the group T{p,p) satisfy 

TaGQ[e^,e^]. □ 

Mostow proves this result in lemma 17.2.1 in [Mos80] using implicitely a trace 
formula similar to our combinatorial trace formula for the special case of the groups 
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11. Applications: Non-Discreteness of Triangle Groups 

In this section wc describe some necessary conditions for a triangle group repre- 
sentation to be a discrete embedding. For the computations in this and the next 
section the parameterisation of complex hyperbolic (pi,p2,P3)-triangle groups by 



a 1 + cos a 

t = ctan — = \ 

2 V 1 — cos a 

is more suitable than the parameterisation by the angular invariant a itself. Recall 
that a (pi,p2,P3)-triangle group exists for t G {—too, too), where 

1 1 + Coo , + 
too = \ --, and Coo = 7, • 

V 1 - Coo 2rir2r3 

(For Coo > 1 resp. Cqo < — 1 wc set too = +oo rcsp. too = — oo.) 

Let (j) : r(pi,p2,P3) PU(2, 1) be a complex hyperbolic triangle group repre- 
sentation and G := (t>(r{pi,P2,P3)) the corresponding complex hyperbolic triangle 
group. Assume that 7 is an element of infinite order in ^{pi,P2-p3) that its 
image (jj^j) in G is regular elliptic. Then there are two cases, either 0(7) is of finite 
order then (p is not injective, or (j){'y) is of infinite order and then (j> is not discrete 
because the subgroup of G generated by (/)(7) is not discrete. In particular, if a 
(pi,P2,P3)-representation is a discrete embedding, then the elements ii23 and 
i'k,k-i,k,k+i are not regular elliptic. 

We have shown that the trace of the element wa = ^3231 is the real number 

(Ts = (16rfr2 + — 1) — 16rir2r3 cos a. 

To check if the clement wa is regular elliptic wc use the discriminant function p 
described in section 2. As explained there, an element A G SU(2, 1) whose trace is 
real is regular elliptic iff traced e (—1,3). The inequality afc > — 1 holds for any 
fc = 1, 2, 3 and a € M except in the case 2rir2 = rs and a G 2^ -Z since 

(Tfc = (16r^_ir^.+i + 4rj. - 1) - 16rir2r3COsa 

> 16r^_ir^_,_i -h 4r^ - 1 - 16rir2r3 

= 4(2rfe_irfe+i - rkf - 1 > -1. 



It remains to resolve the inequality 0-3 < 3 with respect to cos a. We obtain the 
following proposition 

Proposition 12. The element wa = '-3231 in a {pi,P2, Pa) -triangle group is regular 
elliptic iff \t\ > tA, where 



\ + CA , Arlrl + rl-l 
tA = \ and CA = . 

1 - CA 4rir2r3 

(For Ca > 1 resp. ca < —1 we set tA = +00 resp. tA = —00.) 
This proposition implies 

Corollary 13. The {pi,p2,P3) -triangle group representation with \t\ > tA is not a 
discrete embedding, because the element wa is regular elliptic. 

Proposition 14. It holds ca ^ Cqo and tA ^ too • 
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Proof. We obtain 

4rir2r3 • {c^ - ca) ^ rl - {2r\ - l)(2r2 - 1) 

2/7r\ /27r\ /2^\ 

= cos — — COS COS — . 

VPs/ \V\ ) \V2 ) 

For P3 ^ pi , P2 ^ 4 this implies Coo ^ ca because of 

/7r\ /27r\ /27r\ „ 
cos — \^ COS , COS — > 0. 

vpsy vpiy Kvi) 

In the case pi = 3 the computations are straightforward. □ 

Remark. Because of ak — <Tk+i = 4(r^ — r^_|_i)(l — 4r^_;^) it holds cri ^ f72 ^ 0-3. 

This means that among the elements Lk,k-i,k.k+i the clement wa = ',i2,'!i is the 
first one to become elliptic as t > tends to too and therefore there is no need to 
control the ellipticity of the elements ^1312 and ^2123- 

To check the ellipticity of the element wb = '-123 is computationally a much more 
involved task. A tedious but straightforward computation shows that p(trace(wB)) 
is of the form 

p{tmcciwB)) - J^-^' 

where fs is an even polynomial of degree 6. Moreover, it holds /b(0) > for 
''I) ''2, ''3 > 1/2, and the coefficient by vanishes if Cqo = 1. The polynomial /b 
can be computed explicitly, but we shall not use these formulas, so we omit them. 

12. Triangle Groups with + rf + rf = 1 + 2rir2r3 
In this section we study triangle groups corresponding to triples (ri, r2, r^) with 

rl+rl+rl = l + 2rir2r3. {*) 

These triangle groups seem to share a lot of properties with the ideal triangle 
groups. Possibly some of the methods used for ideal case can be used in this case 
too. 

The equation (*) is equivalent to Cqo = 1 and too = +00. This implies that all 
real numbers occur as values of parameter t for some (t^i, y>2, <^3)-triangle in H^. 

We first consider the case of triangle group corresponding to a complex hyperbolic 
triangle with all three vertices in U dH^. In this case = cos ipk, where (pi, 
ip2, and (ps are the angles of the triangle. We assume ri < r2 < r^ and hence 
<^i ^ <^2 ^ </'3 ^ 0. The equation (*) is then equivalent to 

ipi =ip2 + ip3. 

The signature of a non-ideal triangle group which belongs to the family (*) is of 
the form 

{Pi,P2,P3) = {abp, a{a + b)p, h(a + b)p) 
for some positive integer numbers a. b, and p with a ^ fe. Some special cases are 
(p, 2p, 2p)-groups, (2p, 3p, 6p)-groups etc. If we allow also ideal vertices, we obtain 
{p,p, oo)-groups studied by J. Wyss-Gallifent (compare (4) in section 14) and among 
them the ideal triangle groups. A computation shows that 

sin^(yi +(^2) ^ ^ 
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In the case of an ultra-parallel complex hyperbolic ^2, ^3]-triangle group we 
similarly obtain £3 = £i + £2 and 

4/1 

Some special cases are i?, 2^] -groups studied by J. Wyss-Gallifent (compare (5) 
in section 14), [i, 21, 3^]-groups etc. 

The computation of 

fsit) 



p{tTa,ce{wB)) 



for the triples (ri,r2,r3) satisiying the equation (*) is simpler. We obtain 

= (1 - + - SOR^ + IIR + 2)i^ + {loAR^ + 48i?^ + 12i? + 1), 

where R = rir2r3. The polynomial fs has no roots if R < 7/8. For i? > 7/8 the 
roots of the polynomial /b can be computed explicitly as ±t%, where 



t% = 



' 2 -Flli? - 80i?2 + 64i?3 ± R^{%R-7f{%R+l) 



More precisely, 

(1) for R < 7/8 we have /^(t) > for all t e R, 

(2) for R = 7/8 we have fslt) > for all t € ]R\{±4}, 

(3) for 7/8 < i? < 1 we have fsit) > for \t\ < or \t\ > and fsit) < 
for tg < \t\ <t+, 

(4) for i? > 1 we have fsit) > for \t\ < t+ and /^(t) < for \t\ > t+. 
In the ideal triangle group case we have R = 1 and hence (compare [GP92]) 

p(ti23) • (t^ + 1)' = 1024 • (125 - 3*2). 

In the ultra-parallel case the element wa remains hyperbolic for alH e K and 
the element wb goes elliptic for t = t'^, hence 

Proposition 15. Any ultra-parallel [£\,£2,£3\-triangle group such that 

i'\+r'i+rl = l-\- 2rir2r3, 
i.e. any ultra-parallel [^1,^2,^1 -\- £2\-triangle group is of type B. 

In the non-ideal case we obtain the following result 
Proposition 16. A {pi,p2,P3) -triangle group such that 

''i + ^2 + ''s = 1 + 2rir2r3 

and 

13 + \/2()7 
rir2r3 > ^ « 0,9448 

is of type B. For instance, the triples {p,p, 00) with p ^ 14 and the triples {p, 2p, 2p) 
with p^ 12 are of type B. 
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Proof. The element wa goes elliptic for t = Ia- The element wb goes elliptic for 
t = tg since 7/8 < R< 1. We are going to prove that tA> t~^- We have 

and hence ca ^ -R- This implies Ia ^ t% where 

{t\ is the value of Ia for ri = r2 = ^/R and rs = 1.) We obtain 

/B(i^) = -^-(16i?^-13i?-2). 

For >(13 + \/297) /32 it holds 16i?2 - 13i? - 2 > 0. This impUes /b (i a) < and 
hence t\ ^ t^. The inequalities tA^t\> imply tA> f^- □ 

13. Applications: Arithmetic Properties of Traces 

Let G{pi,p2,P3;n) be the (pi,p2,P3)-triangle group such that 13132 is a rotation 
by 27r/n. Then T3132 = 1 + 2cos(27r/n) (compare Lemma 3.18 in [Par]). On the 
other hand 

T3132 = 0-3 = (ler^rl + 4r3 - 1) - 16rir2r3 cos a 

and hence 

cosa = (8rir2r3)'^ • (^{8rjrl + 2r| - 1) - cos . 
Hero n = 00 is allowed and means r3i32 = 3. 

Proposition 17. Let r be the trace of the element in G{pi,p2,P3',n), then 
2Re(T), jrl^eZ 2cos ( — ) ,2cos f — ) ,2cos ( — ) ,2cos ' 



Pi J \P2 J \P3 J \n J _ 

Proof. Because of 2cos(27r/pfe) — 4r^ — 2 we have 

Z [2cos(27r/pi),2cos(27r/p2),2cos(27r/p3),2cos(27r/n)] = i? [2 cos(27r/n)] , 

where R = Z[4rf , 4r|, 4r|]. For the trace r of an element of G(pi,p2,P3; f^) it holds 
according to the combinatorial trace formula r = (—1)1"! • (2 + where \a\ is the 
length of a and 

We have 2Re(r) = (-l)!"! • (4 + 2Re(0) and \t\^ = 4 + 4Re(0 + hence is is 
sufficient to prove that 2Re(^) and |^|^ belong to the ring i?[2cos(27r/n)]. We have 

2Re(0 =^(lw (2cos(a«;)) 

and 

l^i^ = ^ f = II + I^^-I^'-C^ cos(a(w; - w'))). 
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But 2 cos(q;w) is a polynomial of degree \w\ in 2 cos a with integer coefficients, 
even for even \w\ and odd for odd |w| (Chebyshev polynomials, compare for exam- 
ple [San95]). This implies 

2 cos{aw) e (8rir2r3)"l'"l • -R[2cos(27r/n)] 

because of 

2 cos a € (8rir2r3)"^ • ii[2 cos(27r/n)] 

and hence 

(2cosa)l'"l-2J e (Srirara)-!""! • {4rl ■ Ar^ ■ Arjy ■ R[2 cos{2tt / n)] 
C (Snrars)-!'"! •i?[2cos(27r/n)]. 

On the other hand the coefficients satisfy G (8rir2r3)l'"' • R by lemma 8. This 
finishes the proof. □ 

Corollary 18. Let r be the trace of an element in G{pi,p2,P3',ri) with 

{pi,P2,P3,n} C {3,4,6,oo,g}, 

where q is any natural number, then 2Re(r) and |Tp both belong to the ring 
Z[2cos(27r/g)]. 

Proof. For a e {3, 4, 6, oo} we have 2 cos(27r/a) e Z. □ 

14. History and Geography 

In this section wc describe some of the results in the area of complex hyperbolic 
triangle groups and specify there position in the space of triples (ri,r2,r3). 

(1) ideal triangle case: n = r2 = rs = 1, {pk = oo, (pk = 0), type B. 

Ideal triangle group representations were first studied in [GP92] by William 
Goldman and .lohn Parker. They proved that an ideal triangle group represen- 
tation is a discrete embedding for cos ct < j§ and is not a discrete embedding for 
cos a > 1^, and they conjectured that an ideal triangle group representation is 
a discrete embedding for cos a ^ This conjecture was confirmed by Richard 
E. Schwartz in [SchOlb]. He proved that an ideal triangle group representation 
is a discrete embedding if and only if cos a < In [SchOla] he also used the 
so called last ideal triangle group, i.e. the group with ti23 parabolic, i.e. with 
cos a = |j, to construct the first example of a complete hyperbolic 3-manifold 
with a spherical CR-structure. Hanna Sandler suggested in [San95] to study 
formulas for the traces of elements in an ideal triangle group in order to prove 
the Goldman-Parker conjecture. 

(2) (4, 4, 4)-groups: n = r2 = rs = 2-1/2, = 4, = 7^/4), type A. 

Richard E. Schwartz studied in [Sch03a] the groups G(4,4,4;n) and used the 
group G(4,4,4;7), i.e. the (4, 4, 4)-group with cos a = 2-^/^ • (2-cos(27r/7)) to 
construct the first example of a compact hyperbolic S-manifold with a spherical 
CR-structure. 
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(3) (4, 4, (X))-groups: n — r2 — 2 rs = 1, type A. 

Justin Wyss-Gallifent studied (4, 4,oo)-groups in [WGOO]. He proved that a 
(4, 4, oc)-group is not discrete for cos a > 1/2 and that the groups G{A, 4, oo; n) 

arc discrete for n = 3, 4, oo. 

(4) (p.p. oc)-groups: ri = r2, = 1. 

The {p,p, oo)-groups were studied also by Justin Wyss-GaUifent in [WGOO]. He 
proved that a triple oo) is of type A for p ^ 13 and of type B for p > 14. 

(5) ultra-parallel 2^]-groups: ri = r2 = r, rs = 2r^ — 1. 

Justin Wyss-GaUifent studied in [WGOO] ultra-parallel [I, I, 2£]-groups and ob- 
tained partial results similar to the results in [GP92] on ideal triangle groups. 

(6) Mostow's non-arithmetic complex hyperbolic lattices: /Ui = /U2 = /U3 = e p , 
p e {3,4, 5}, n = r2 = ra = a=^ + ^-f,peZ. 

G. D. Mostow studied in [Mos80] some triangle groups generated by three ^- 
reflections of the same order p G {3, 4, 5} in the sides of a equiangular triangle 
with angles related to the order p and with special values of the angular invari- 
ant a and obtained the first examples of non-arithmetic complex hyperbolic 
lattices. 
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